---
id: evaluation-arena-test-cases
title: Arena Test Case
sidebar_label: Arena
---

## Quick Summary

An **arena test case** is a blueprint provided by `deepeval` for you to compare which iteration of your LLM app performed better. It works by comparing each contestants's `LLMTestCase` to run comparisons, and currently only supports the `LLMTestCase` for single-turn, text-based comparisons.

:::info
Support for `ConversationalTestCase` and `MLLMTestCase` is coming soon.
:::

The `ArenaTestCase` currently only runs with the `ArenaGEval` metric, and all that is required is to provide a list of `Contestant`s:

```python title="main.py"
from deepeval.test_case import ArenaTestCase, LLMTestCase, Contestant

test_case = ArenaTestCase(contestants=[
    Contestant(
        name="GPT-4",
        hyperparameters={"model": "gpt-4"},
        test_case=LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris",
        ),
    ),
    Contestant(
        name="Claude-4",
        hyperparameters={"model": "claude-4"},
        test_case=LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris is the capital of France.",
        ),
    ),
    Contestant(
        name="Gemini-2.5",
        hyperparameters={"model": "gemini-2.5-flash"},
        test_case=LLMTestCase(
            input="What is the capital of France?",
            actual_output="Absolutely! The capital of France is Paris 😊",
        ),
    ),
])
```

Note that all `input`s and `expected_output`s you provide across contestants **MUST** match.

:::tip
For those wondering why we took the choice to include multiple duplicated `input`s in `LLMTestCase` instead of moving it to the `ArenaTestCase` class, it is because an `LLMTestCase` integrates nicely with the existing ecosystem.

You also shouldn't worry about unexpected errors because `deepeval` will throw an error if `input`s or `expected_output`s aren't matching.
:::

## Arena Test Case

The `ArenaTestCase` takes a simple `contestants` argument, which is a list of `Contestant`s.

```python
contestant_1 = Contestant(
    name="GPT-4",
    hyperparameters={"model": "gpt-4"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris",
    ),
)

contestant_2 = Contestant(
    name="Claude-4",
    hyperparameters={"model": "claude-4"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris is the capital of France.",
    ),
)

contestant_3 = Contestant(
    name="Gemini-2.5",
    hyperparameters={"model": "gemini-2.5-flash"},
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Absolutely! The capital of France is Paris 😊",
    ),
)

test_case = ArenaTestCase(contestants=[contestant_1, contestant_2, contestant_3])
```

### Contestant

A `Contestant` represents a single unit of [llm interaction](/docs/evaluation-test-cases#what-is-an-llm-interaction) from a specific version of your LLM app. It accepts a `test_case`, a `name` to identify the LLM app version that was used to generate the test case, and optionally any `hyperparameters` associated with the LLM version.

```python
from deepeval.test_case import Contestant, LLMTestCase
from deepeval.prompt import Prompt

contestant_1 = Contestant(
    name="GPT-4",
    test_case=LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris",
    ),
    hyperparameters={
        "model": "gpt-4",
        "prompt": Prompt(alias="test_prompt", text_template="You are a helpful assistant."),
    },
)
```

## Using Test Cases For Evals

The [`ArenaGEval` metric](/docs/metrics-arena-g-eval) is the only metric that uses an `ArenaTestCase`, which picks a "winner" out of the list of contestants:

```python
from deepeval.metrics import ArenaTestCase, LLMTestCaseParams
...

arena_geval = ArenaGEval(
    name="Friendly",
    criteria="Choose the winner of the more friendly contestant based on the input and actual output",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
)

compare(test_cases=[test_case], metric=arena_geval)
```

The `ArenaTestCase` streamlines the evaluation by automatically masking contestant names (to ensure unbiased judging) and randomizing their order.
